{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "b7b14d57-68cc-4beb-9b99-76827687db88",
   "metadata": {},
   "source": [
    "(dataset-preparation)=\n",
    "## Prepare your test dataset\n",
    "\n",
    "Evaluating any ML pipeline will require several data points that constitutes a test dataset. For Ragas, the data points required for evaluating your RAG completely are\n",
    "\n",
    "- `question`: A question or query that is relevant to your RAG.\n",
    "- `contexts`: The retrieved contexts corresponding to each question. This is a `list[list]` since each question can retrieve multiple text chunks.\n",
    "- `answer`:  The answer generated by your RAG corresponding to each question.\n",
    "- `ground_truth`: The expected correct answer corresponding to each question.\n",
    "\n",
    "For the purpose of this notebook, I have this dataset prepared from a simple RAG that I created myself to help me with NLP research. Let's use it."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "326f3dc1-775f-4ec5-8f27-afd76e9b5b22",
   "metadata": {},
   "outputs": [],
   "source": [
    "from datasets import load_dataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "1ee49bc5-4661-4435-8463-197877c18fa3",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "715a428e3fe14e0996b699fb22b54382",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Downloading readme:   0%|          | 0.00/28.0 [00:00<?, ?B/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "13736c6c8cb841c2a989a9b67a20e627",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Downloading data files:   0%|          | 0/1 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "2f02e071aa474da0aa78e251ba3fe125",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Downloading data:   0%|          | 0.00/11.0k [00:00<?, ?B/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "a81e14abccbe41eaa40b0787f6a0d15b",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Extracting data files:   0%|          | 0/1 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "d6234eb22ffe4fe1bdfc33f6e5411e83",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Generating test split: 0 examples [00:00, ? examples/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>question</th>\n",
       "      <th>ground_truth</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>How does instruction tuning affect the zero-sh...</td>\n",
       "      <td>For larger models on the order of 100B paramet...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>What is the Zero-shot-CoT method and how does ...</td>\n",
       "      <td>Zero-shot-CoT is a zero-shot template-based pr...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>How does prompt tuning affect model performanc...</td>\n",
       "      <td>Prompt tuning improves model performance in im...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>What is the purpose of instruction tuning in l...</td>\n",
       "      <td>The purpose of instruction tuning in language ...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>What distinguishes Zero-shot-CoT from Few-shot...</td>\n",
       "      <td>Zero-shot-CoT differs from Few-shot-CoT in tha...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                            question  \\\n",
       "0  How does instruction tuning affect the zero-sh...   \n",
       "1  What is the Zero-shot-CoT method and how does ...   \n",
       "2  How does prompt tuning affect model performanc...   \n",
       "3  What is the purpose of instruction tuning in l...   \n",
       "4  What distinguishes Zero-shot-CoT from Few-shot...   \n",
       "\n",
       "                                        ground_truth  \n",
       "0  For larger models on the order of 100B paramet...  \n",
       "1  Zero-shot-CoT is a zero-shot template-based pr...  \n",
       "2  Prompt tuning improves model performance in im...  \n",
       "3  The purpose of instruction tuning in language ...  \n",
       "4  Zero-shot-CoT differs from Few-shot-CoT in tha...  "
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "eval_dataset = load_dataset(\"explodinggradients/prompt-engineering-guide-papers\")\n",
    "eval_dataset = eval_dataset[\"test\"].to_pandas()\n",
    "eval_dataset.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "84ae0719-82bc-4103-8299-3df7021951e1",
   "metadata": {},
   "source": [
    "As you can see, the dataset contains two of the required attributes mentioned,that is `question` and `ground_truth` answers. Now we can move on our next step to collect the other two attributes.\n",
    "\n",
    ":::{note}\n",
    "*We know that it's hard to formulate a test data containing Question and ground truth answer pairs when starting out. We have the perfect solution for this in this form of a ragas synthetic test data generation feature. The questions and ground truth answers were created by [ragas synthetic data generation](./testset_generation.md) feature. Check it out here once you finish this notebook*\n",
    ":::"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6184b6b5-7373-4665-9754-b4fc08929000",
   "metadata": {},
   "source": [
    "#### Simple RAG pipeline\n",
    "\n",
    "Now with the above step we have two attributes needed for evaluation, that is `question` and `ground_truth` answers. We now need to feed these test questions to our RAG pipeline to collect the other two attributes, ie `contexts` and `answer`.  Let's build a simple RAG using llama-index to do that. \n",
    "\n",
    ":::{note}\n",
    "I'm also using a sample corpus containing NLP papers and open-ai models for building the RAG pipeline. You should be running the same through your RAG pipeline. This is purely for demonstration purposes. And I assume that if you're here you already have a RAG pipeline ready to use.\n",
    ":::"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "e4b5136c",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Cloning into 'prompt-engineering-guide-papers'...\n",
      "remote: Enumerating objects: 19, done.\u001b[K\n",
      "remote: Counting objects: 100% (15/15), done.\u001b[K\n",
      "remote: Compressing objects: 100% (15/15), done.\u001b[K\n",
      "remote: Total 19 (delta 1), reused 0 (delta 0), pack-reused 4\u001b[K\n",
      "Unpacking objects: 100% (19/19), 3.07 MiB | 6.46 MiB/s, done.\n",
      "Filtering content: 100% (3/3), 18.03 MiB | 6.34 MiB/s, done.\n"
     ]
    }
   ],
   "source": [
    "! git clone https://huggingface.co/datasets/explodinggradients/prompt-engineering-guide-papers"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "id": "3aa590ae",
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "\n",
    "PATH = \"./prompt-engineering-guide-papers\"\n",
    "os.environ[\"OPENAI_API_KEY\"] = \"your-open-ai-key\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "d7bbceb1-5e05-422d-8690-f49fc71245d4",
   "metadata": {},
   "outputs": [],
   "source": [
    "import nest_asyncio\n",
    "from llama_index.core.indices import VectorStoreIndex\n",
    "from llama_index.core.readers import SimpleDirectoryReader\n",
    "from llama_index.core.service_context import ServiceContext\n",
    "from datasets import Dataset\n",
    "\n",
    "nest_asyncio.apply()\n",
    "\n",
    "\n",
    "def build_query_engine(documents):\n",
    "    vector_index = VectorStoreIndex.from_documents(\n",
    "        documents,\n",
    "        service_context=ServiceContext.from_defaults(chunk_size=512),\n",
    "    )\n",
    "\n",
    "    query_engine = vector_index.as_query_engine(similarity_top_k=3)\n",
    "    return query_engine\n",
    "\n",
    "\n",
    "# Function to evaluate as Llama index does not support async evaluation for HFInference API\n",
    "def generate_responses(query_engine, test_questions, test_answers):\n",
    "    responses = [query_engine.query(q) for q in test_questions]\n",
    "\n",
    "    answers = []\n",
    "    contexts = []\n",
    "    for r in responses:\n",
    "        answers.append(r.response)\n",
    "        contexts.append([c.node.get_content() for c in r.source_nodes])\n",
    "    dataset_dict = {\n",
    "        \"question\": test_questions,\n",
    "        \"answer\": answers,\n",
    "        \"contexts\": contexts,\n",
    "    }\n",
    "    if test_answers is not None:\n",
    "        dataset_dict[\"ground_truth\"] = test_answers\n",
    "    ds = Dataset.from_dict(dataset_dict)\n",
    "    return ds"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "d299259a-1064-44d4-9c96-5ae423d9e2f8",
   "metadata": {},
   "outputs": [],
   "source": [
    "reader = SimpleDirectoryReader(PATH, num_files_limit=30, required_exts=[\".pdf\"])\n",
    "documents = reader.load_data()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "546f7b5c-36b7-4ffc-90d6-bea27df01aa5",
   "metadata": {},
   "outputs": [],
   "source": [
    "test_questions = eval_dataset[\"question\"].values.tolist()\n",
    "test_answers = eval_dataset[\"ground_truth\"].values.tolist()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "id": "52a67da6-2eeb-452c-a90c-5c1ea860545b",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/var/folders/ww/sk5dkfhn673234cmy5w7008r0000gn/T/ipykernel_20117/2872893575.py:12: DeprecationWarning: Call to deprecated class method from_defaults. (ServiceContext is deprecated, please use `llama_index.settings.Settings` instead.) -- Deprecated since version 0.10.0.\n",
      "  documents, service_context=ServiceContext.from_defaults(chunk_size=512),\n"
     ]
    }
   ],
   "source": [
    "query_engine1 = build_query_engine(documents)\n",
    "result_ds = generate_responses(query_engine1, test_questions, test_answers)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "id": "031e8006",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Dataset({\n",
       "    features: ['question', 'answer', 'contexts', 'ground_truth'],\n",
       "    num_rows: 20\n",
       "})"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "result_ds"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "84f37875",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "5ae9db58e6c6466f9a1b10f235cbbba3",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Downloading data:   0%|          | 0.00/126k [00:00<?, ?B/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "e60e915d1eee45a5a0958a4f71dc23f5",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Generating test split: 0 examples [00:00, ? examples/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>question</th>\n",
       "      <th>ground_truth</th>\n",
       "      <th>answer</th>\n",
       "      <th>contexts</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>How does instruction tuning affect the zero-sh...</td>\n",
       "      <td>For larger models on the order of 100B paramet...</td>\n",
       "      <td>For larger models with around 100B parameters,...</td>\n",
       "      <td>[Published as a conference paper at ICLR 2022\\...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>What is the Zero-shot-CoT method and how does ...</td>\n",
       "      <td>Zero-shot-CoT is a zero-shot template-based pr...</td>\n",
       "      <td>The Zero-shot-CoT method is a zero-shot templa...</td>\n",
       "      <td>[Similar to\\nFew-shot-CoT, Zero-shot-CoT facil...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>How does prompt tuning affect model performanc...</td>\n",
       "      <td>Prompt tuning improves model performance in im...</td>\n",
       "      <td>Prompt tuning has been shown to enhance model ...</td>\n",
       "      <td>[The orange bars indicate standard deviation a...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>What is the purpose of instruction tuning in l...</td>\n",
       "      <td>The purpose of instruction tuning in language ...</td>\n",
       "      <td>The purpose of instruction tuning in language ...</td>\n",
       "      <td>[Although one might\\nexpect labeled data to ha...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>What distinguishes Zero-shot-CoT from Few-shot...</td>\n",
       "      <td>Zero-shot-CoT differs from Few-shot-CoT in tha...</td>\n",
       "      <td>Zero-shot-CoT requires prompting LLMs twice bu...</td>\n",
       "      <td>[Baselines We compare our Zero-shot-CoT mainly...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                            question  \\\n",
       "0  How does instruction tuning affect the zero-sh...   \n",
       "1  What is the Zero-shot-CoT method and how does ...   \n",
       "2  How does prompt tuning affect model performanc...   \n",
       "3  What is the purpose of instruction tuning in l...   \n",
       "4  What distinguishes Zero-shot-CoT from Few-shot...   \n",
       "\n",
       "                                        ground_truth  \\\n",
       "0  For larger models on the order of 100B paramet...   \n",
       "1  Zero-shot-CoT is a zero-shot template-based pr...   \n",
       "2  Prompt tuning improves model performance in im...   \n",
       "3  The purpose of instruction tuning in language ...   \n",
       "4  Zero-shot-CoT differs from Few-shot-CoT in tha...   \n",
       "\n",
       "                                              answer  \\\n",
       "0  For larger models with around 100B parameters,...   \n",
       "1  The Zero-shot-CoT method is a zero-shot templa...   \n",
       "2  Prompt tuning has been shown to enhance model ...   \n",
       "3  The purpose of instruction tuning in language ...   \n",
       "4  Zero-shot-CoT requires prompting LLMs twice bu...   \n",
       "\n",
       "                                            contexts  \n",
       "0  [Published as a conference paper at ICLR 2022\\...  \n",
       "1  [Similar to\\nFew-shot-CoT, Zero-shot-CoT facil...  \n",
       "2  [The orange bars indicate standard deviation a...  \n",
       "3  [Although one might\\nexpect labeled data to ha...  \n",
       "4  [Baselines We compare our Zero-shot-CoT mainly...  "
      ]
     },
     "execution_count": 1,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "result_ds.to_pandas().head()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "50c9bbf9",
   "metadata": {},
   "source": [
    "Done. You now have the dataset required for evaluating your RAG system. Let's move on to the next step. That's the actual evaluation of your RAG system. Checkout evaluation [here](./evaluation.md)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c020fec7-8451-46ae-8b2d-192ae468428e",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "ragas",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.8"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}
